Platform Behavior under Market Shocks: A Simulation Framework and Reinforcement-Learning Based Study

We study the behavior of an economic platform (e.g., Amazon, Uber Eats, Instacart) under shocks, such as COVID-19 lockdowns, and the effect of different regulation considerations. To this end, we develop a multi-agent simulation environment of a platform economy in a multi-period setting where shocks may occur and disrupt the economy. Buyers and sellers are heterogeneous and modeled as economically-motivated agents, choosing whether or not to pay fees to access the platform. We use deep reinforcement learning to model the fee-setting and matching behavior of the platform, and consider two major types of regulation frameworks: (1) taxation policies and (2) platform fee restrictions. We offer a number of simulated experiments that cover different market settings and shed light on regulatory tradeoffs. Our results show that while many interventions are ineffective with a sophisticated platform actor, we identify a particular kind of regulation - fixing fees to the optimal, no-shock fees while still allowing a platform to choose how to match buyers and sellers - as holding promise for promoting the efficiency and resilience of the economic system. © 2023 ACM.

Reinforcement Learning-based Counter-Misinformation Response Generation: A Case Study of COVID-19 Vaccine Misinformation

The spread of online misinformation threatens public health, democracy, and the broader society. While professional fact-checkers form the first line of defense by fact-checking popular false claims, they do not engage directly in conversations with misinformation spreaders. On the other hand, non-expert ordinary users act as eyes-on-the-ground who proactively counter misinformation - recent research has shown that 96% counter-misinformation responses are made by ordinary users. However, research also found that 2/3 times, these responses are rude and lack evidence. This work seeks to create a counter-misinformation response generation model to empower users to effectively correct misinformation. This objective is challenging due to the absence of datasets containing ground-truth of ideal counter-misinformation responses, and the lack of models that can generate responses backed by communication theories. In this work, we create two novel datasets of misinformation and counter-misinformation response pairs from in-the-wild social media and crowdsourcing from college-educated students. We annotate the collected data to distinguish poor from ideal responses that are factual, polite, and refute misinformation. We propose MisinfoCorrect, a reinforcement learning-based framework that learns to generate counter-misinformation responses for an input misinformation post. The model rewards the generator to increase the politeness, factuality, and refutation attitude while retaining text fluency and relevancy. Quantitative and qualitative evaluation shows that our model outperforms several baselines by generating high-quality counter-responses. This work illustrates the promise of generative text models for social good - here, to help create a safe and reliable information ecosystem. The code and data is accessible on https://github.com/claws-lab/MisinfoCorrect. © 2023 Owner/Author.

A Deep Learning-Based Preventive Measures of COVID-19 in a crowd using Reinforcement Model over GAN for Enhanced efficiency

The fast human-to-human spread of COVID-19 has caused significant lifestyle changes for many individuals. At the end of January 2020, the pandemic began, and many nations responded with varying degrees of testing, sanitation, lockdown, and quarantine centers. New normals of testing, sanitization, social separation, and lockdown are being implemented, and people are gradually returning to work and other daily routines. The COVID-19 infected population is monitored by testing individuals regularly. But it's a resource-heavy endeavor to test everyone without good reason. An optimum strategy is required to efficiently identify persons who are most likely to test positive for COVID-19. Sanitation is utilized for both persons and public spaces to eliminate germs. However, the disruption of governmental operations and economic development makes the use of lockdown and quarantine centers a resource-intensive endeavor. Conversely, it degrades the standard of living across a society. Furthermore, keeping people inside their houses or quarantine centers for an unlimited amount of time would not allow the government to care for everyone. These variables impact virus propagation, human health and happiness, available resources, and the economy's health, making their management resource-intensive. counting and density estimation are both attempts to create clever and efficient algorithms that can interpret the data provided by images to carry out Efficiency. GANs have been proven to have promising applications in overcoming the data dearth problem in COVID-19 lung image analysis. The Convolutional Neural Network (CNN) models built for the diagnosis of COVID-19 have benefited from the GAN-generated data used to refine their training. Moreover, GANs have helped improve the performance of CNNs by super-resolving pictures and performing segmentation. This work highlights the Reinforcement deep learning model over the fundamental constraints of the possible transformation of GANs-based approaches. This work proposes the model be developed with a new intelligent approach using RL to quantify these different types of testing considered for social distancing, face mask detection, limiting the gathering, and locking the location using the Q Learning technique. Different RL algorithms are implemented, and agents are equipped with these algorithms so that they may interact with the environment and learn the optimum method for doing so. © 2023 IEEE.

Enhancing Immersive User Experience Quality of StudoBot Telepresence Robots with Reinforcement Learning

The pandemic situation (Covid 19) brought new challenges in the education sector while simultaneously presenting unique opportunities for technology enabled services. The use of Mobile Robotic Telepresence systems in educational sector is promising as it provides means to significantly enhance the involvement and benefits to stakeholders involved in such interactions. An immersive user interaction with such a system depends on many aspects which are both static and dynamic. We approach the dynamic aspect of such interactions recognizing that the video and audio aspects of such a system will require fine tuning and adaptation. Closely related is the aspect of maintaining the necessary quality of network connection. Considering each of these aspects a reinforcement learning mechanism is incorporated to improve the overall user experience with such a system. A working system is built and experiments performed to demonstrate the effectiveness of the approach. Reward generation matrix, a crucial piece of data gathering from the environment, takes about 45 minutes, offline training time is less than a second, while the robot is able to cover the workspace in slightly less than a minute. The system is not limited to educational sector alone and provides a foundational framework to extend the concepts and principles to adjacent markets. © 2023 IEEE.

Computer-Aided System for COVID-19 Using Semi-supervised-based Ensemble Learning and Reinforcement Learning

Coronavirus Disease 2019(COVID-19) has shocked the world with its rapid spread and enormous threat to life and has continued up to the present. In this paper, a computer-aided system is proposed to detect infections and predict the disease progression of COVID-19. A high-quality CT scan database labeled with time-stamps and clinicopathologic variables is constructed to provide data support. To our knowledge, it is the only database with time relevance in the community. An object detection model is then trained to annotate infected regions. Using those regions, we detect the infections using a model with semi-supervised-based ensemble learning and predict the disease progression depending on reinforcement learning. We achieve an mAP of 0.92 for object detection. The accuracy for detecting infections is 98.46%, with a sensitivity of 97.68%, a specificity of 99.24%, and an AUC of 0.987. Significantly, the accuracy of predicting disease progression is 90.32% according to the timeline. It is a state-of-the-art result and can be used for clinical usage. © 2022 IEEE.

Evaluation of Masked Face Recognition of FaceNet Implemented with Machine Learning Algorithms

In image processing, Convolutional Neural Network (CNN) is an important tool for isolating image attributes for using with applications such as facial recognition. According to an outbreak of COVID-19, wearing masks has made face recognition less effective since face details are covered. FaceNet platform is a face feature extraction that is commonly applied to classification applications. Those applications embed FaceNet platform with supervised learning machine learning types to classify the considered objected on the detected image. Recently, Reinforcement Learning (RL) has been used in many applications on both prediction and classification tasks. However, the learning efficiency of RL has not been implemented and evaluated on masked face recognition yet. Therefore, the efficiency of the supervised learning techniques, ANN, KNN and SVM, are also implemented with the FaceNet platform for masked face recognition and they are compared with FaceNet platform implemented with the RL. The simulation results showed that ANN is the most efficient technique and followed by RL, KNN and SVM. The difference in efficiency (F1-scroce) between RL and the neural network was only 2%, but RL took four times more training time. © 2022 IEEE.

Reduce Cheating in e-Exams Using Machine Learning: State of the Art

Cheating on online exams becomes a black spot in distance learning environments. On the one hand, it threatens the credibility of these exams by violating the principle of equality and success on merit. On the other hand, it also has negative repercussions on the reputation of the institutions. Without a doubt, in the Covid-19 health crisis and following the recommendations of the World Health Organization to respect social distancing, the majority of establishments have adopted the distance learning system, including online exams. However, the difficulty of monitoring learner activity in remote settings characterizes this type of assessment by inequity. In practice, each establishment has relied on a monitoring solution adapted according to certain criteria in order to guarantee a fair passage of the exams and to control them well. AI-assisted proctoring tools add a layer of protection to online exams. In this article we will discuss and compare the different uses of Artificial Intelligence tools to reduce cheating in online exams, based on the use of Machine Learning techniques. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Contact Tracing and Epidemic Intervention via Deep Reinforcement Learning

The recent outbreak of COVID-19 poses a serious threat to people's lives. Epidemic control strategies have also caused damage to the economy by cutting off humans' daily commute. In this article, we develop an Individual-based Reinforcement Learning Epidemic Control Agent (IDRLECA) to search for smart epidemic control strategies that can simultaneously minimize infections and the cost of mobility intervention. IDRLECA first hires an infection probability model to calculate the current infection probability of each individual. Then, the infection probabilities together with individuals' health status and movement information are fed to a novel GNN to estimate the spread of the virus through human contacts. The estimated risks are used to further support an RL agent to select individual-level epidemic-control actions. The training of IDRLECA is guided by a specially designed reward function considering both the cost of mobility intervention and the effectiveness of epidemic control. Moreover, we design a constraint for control-action selection that eases its difficulty and further improve exploring efficiency. Extensive experimental results demonstrate that IDRLECA can suppress infections at a very low level and retain more than 95% of human mobility. © 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Hierarchical Multi-agent Model for Reinforced Medical Resource Allocation with Imperfect Information

With the advent of the COVID-19 pandemic, the shortage in medical resources became increasingly more evident. Therefore, efficient strategies for medical resource allocation are urgently needed. However, conventional rule-based methods employed by public health experts have limited capability in dealing with the complex and dynamic pandemic-spreading situation. In addition, model-based optimization methods such as dynamic programming (DP) fail to work since we cannot obtain a precise model in real-world situations most of the time. Model-free reinforcement learning (RL) is a powerful tool for decision-making;however, three key challenges exist in solving this problem via RL: (1) complex situations and countless choices for decision-making in the real world;(2) imperfect information due to the latency of pandemic spreading;and (3) limitations on conducting experiments in the real world since we cannot set up pandemic outbreaks arbitrarily. In this article, we propose a hierarchical RL framework with several specially designed components. We design a decomposed action space with a corresponding training algorithm to deal with the countless choices, ensuring efficient and real-time strategies. We design a recurrent neural network-based framework to utilize the imperfect information obtained from the environment. We also design a multi-agent voting method, which modifies the decision-making process considering the randomness during model training and, thus, improves the performance. We build a pandemic-spreading simulator based on real-world data, serving as the experimental platform. We then conduct extensive experiments. The results show that our method outperforms all baselines, which reduces infections and deaths by 14.25% on average without the multi-agent voting method and up to 15.44% with it. © 2022 Association for Computing Machinery.

Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning

Optimising HVAC operations towards human wellness and energy efficiency is a major challenge for smart facilities management, especially amid COVID situations. Although IoT sensors and deep learning were applied to support HVAC operations, the loss of forecasting accuracy in recursive prediction largely hinders their applications. This study presents a data-driven predictive control method with time-series forecasting (TSF) and reinforcement learning (RL), to examine various sensor metadata for HVAC system optimisation. This involves the development and validation of 16 Long Short-Term Memory (LSTM) based architectures with bi-directional processing, convolution, and attention mechanisms. The TSF models are comprehensively evaluated under independent, short-term recursive, and long-term recursive prediction scenarios. The optimal TSF models are integrated with a Soft Actor-Critic RL agent to analyse sensor metadata and optimise HVAC operations, achieving 17.4% energy savings and 16.9% thermal comfort improvement in the surrogate environment. The results show that recursive prediction leads to a significant reduction in model accuracy, and the effect is more pronounced in the temperature-humidity prediction model. The attention mechanism significantly improves prediction performance in both recursive and independent prediction scenarios. This study contributes new data-driven methods for smart HVAC operations in IoT-enabled intelligent buildings towards a human-centric built environment. © 2023 The Authors

Digital Twin for Transportation Big Data: A Reinforcement Learning-Based Network Traffic Prediction Approach

Vehicular Ad-Hoc Networks (VANETs), as the crucial support of Intelligent Transportation Systems (ITS), have received great attention in recent years. With the rapid development of VANETs, various services have generated a great deal of data that can be used for transportation planning and safe driving. Especially, with the advent of Coronavirus Disease 2019 (COVID-19), the transportation system has been impacted, thus novel modes of transportation planning and intelligent applications are necessary. Digital twins can provide powerful support for artificial intelligence applications in Transportation Big Data (TBD). The features of VANETs are varying, which arises the main challenge of digital twins applying in TBD. Network traffic prediction, as part of digital twins, is useful for network management and security in VANETs, such as network planning and anomaly detection. This paper proposes a network traffic prediction algorithm aiming at time-varying traffic flows with a large number of fluctuations. This algorithm combines Deep Q-Learning (DQN) and Generative Adversarial Networks (GAN) for network traffic feature extraction. DQN is leveraged to carry out network traffic prediction, in which GAN is involved to represent Q-network. Meanwhile, the generative network can increase the number of samples to improve the prediction error. We evaluate the performance of our method by implementing it on three real network traffic data sets. Finally, we compare the two state-of-the-art competing methods with our method. IEEE

Automatic Parameter Tuning via Reinforcement Learning for Crowd Simulation with Social Distancing

Reinforcement learning (RL) has been applied to a variety of fields such as gaming and robot navigation. We study the application of RL in crowd simulation by proposing an automatic parameter tuning system based on Proximal Policy Optimization (PPO). The system can be used with any crowd simulation software to improve the quality of the simulation by automatically assigning parameters to each agent during the simulation. Our experiments indicate that the automatic parameter tuning system can reduce unexpected congestions in counterflow scenarios. In addition, by utilizing the improved commonly used crowd simulation algorithms and our parame-ter tunning system, we can represent social distancing behavior of pedestrians under COVID-19, where pedestrians comply to the suggested social distance when they have enough space to move while they reduce their social distances to others when there is limited space. © 2022 IEEE.

Deep Reinforcement Learning based Haptic Enhancement for Tele-Diagnosis

Tele-Diagnosis is beneficial for medical care in areas with inadequate resources, which helps control the spread of Covid-19 in the current pandemic. Most teleoperated diagnostics are dependent on humans, possibly leading to cognitive issue caused by distanced communication. In this paper, we propose a local haptic enhancement framework to facilitate the remote palpation. The deep deterministic policy gradient (DDPG) algorithm is exploited to compensate for signal transmission due to latency, allowing human to operate without the sense of delay. With the help of weighted recursive least squares (WRLS) method, the interactive force can be estimated on the patient's side despite the lack of force sensors. Fuzzy inference is used to diagnose and classify the extent of disease based on the estimated force and motion state on the remote side, thereby leveraging the remote sensory information to conduct autonomous reasoning. Finally, the final diagnosis is derived by performing minimum risk Bayesian decision based on local and remote inference results. Comparative simulation results have validated the superior performances of the proposed scheme. © 2022 IEEE.

Dynamic Inventory Replenishment with Reinforcement Learning in Managing E-Fulfilment Centres

The demand of retail e-commerce has been rapidly growing due to the digitalization and the COVID-19 pandemic, and thus, the stress on e-fulfilment services continues to increase nowadays. To fulfil daily customers’ orders, effective inventory replenishment is of the essence in order to strike a balance between inventory management costs and service level. This paper describes an enhanced inventory replenishment approach by using reinforcement learning to deal with non-stationary and uncertain demand from customers. The proposed approach relaxes the assumption of stationary demand distribution considered in typical inventory models. Conventional policies derived from such models cannot guarantee optimal re-order quantities, when demand distribution is non-stationary over time. Consequently, reinforcement learning is adopted in the proposed approach to improve feasible solutions continuously in a dynamic business environment. In comparison to the conventional base stock policy, our proposed approach provides cost saving opportunities ranging from 28.5 to 41.3% in a simulated environment. It is found that the value of using data-driven solution approaches to deal with the practical inventory management problem is effective. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Optimizing Pharmaceutical and Non-pharmaceutical Interventions During Epidemics

Controlling the spread of infectious diseases is a major challenge. Understanding the dynamics between human behavior and the spread of infection is essential for policymakers. Evolving contagion dynamics make it difficult to develop an efficient mitigation strategy. In this paper, we develop an epidemiological model to forecast the epidemic and use an offline reinforcement learning framework that adapts to the evolving dynamics of disease spread to optimize the mitigation strategy. We demonstrate that our framework can produce efficient mitigation strategies for the COVID-19 pandemic based on data collected from New York, USA. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

RL Based Unsupervised Video Summarization Framework for Ultrasound Imaging

The need for summarizing long medical scan videos for automatic triage in Emergency Departments and transmission of the summarized videos for telemedicine has gained significance during the COVID-19 pandemic. However, supervised learning schemes for summarizing videos are infeasible as manual labeling of scans for large datasets is impractical by frontline clinicians. This work presents a methodology to summarize ultrasound videos using completely unsupervised learning schemes and is validated on Lung Ultrasound videos. A Convolutional Autoencoder and a Transformer decoder is trained in an unsupervised reinforcement learning setup i.e., without supervised labels in the whole workflow. Novel precision and recall computation for ultrasound videos is also presented employing which high Precision and F1 scores of 64.36% and 35.87% with an average video compression rate of 78% is obtained when validated against clinically annotated cases. Even though demonstrated using lung ultrasound videos, our approach can be readily extended to other imaging modalities. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Reinforcement Learning Enhances the Experts: Large-scale COVID-19 Vaccine Allocation with Multi-factor Contact Network

In the fight against the COVID-19 pandemic, vaccines are the most critical resource but are still in short supply around the world. Therefore, efficient vaccine allocation strategies are urgently called for, especially in large-scale metropolis where uneven health risk is manifested in nearby neighborhoods. However, there exist several key challenges in solving this problem: (1) great complexity in the large scale scenario adds to the difficulty in experts' vaccine allocation decision making;(2) heterogeneous information from all aspects in the metropolis' contact network makes information utilization difficult in decision making;(3) when utilizing the strong decision-making ability of reinforcement learning (RL) to solve the problem, poor explainability limits the credibility of the RL strategies. In this paper, we propose a reinforcement learning enhanced experts method. We deal with the great complexity via a specially designed algorithm aggregating blocks in the metropolis into communities and we hierarchically integrate RL among the communities and experts solution within each community. We design a self-supervised contact network representation algorithm to fuse the heterogeneous information for efficient vaccine allocation decision making. We conduct extensive experiments in three metropolis with real-world data and prove that our method outperforms the best baseline, reducing 9.01% infections and 12.27% deaths.We further demonstrate the explainability of the RL model, adding to its credibility and also enlightening the experts in turn. © 2022 Owner/Author.

Precise Mobility Intervention for Epidemic Control Using Unobservable Information via Deep Reinforcement Learning

To control the outbreak of COVID-19, efficient individual mobility intervention for EPidemic Control (EPC) strategies are of great importance, which cut off the contact among people at epidemic risks and reduce infections by intervening the mobility of individuals. Reinforcement Learning (RL) is powerful for decision making, however, there are two major challenges in developing an RL-based EPC strategy: (1) the unobservable information about asymptomatic infections in the incubation period makes it difficult for RL's decision-making, and (2) the delayed rewards for RL causes the deficiency of RL learning. Since the results of EPC are reflected in both daily infections (including unobservable asymptomatic infections) and long-term cumulative cases of COVID-19, it is quite daunting to design an RL model for precise mobility intervention. In this paper, we propose a Variational hiErarcHICal reinforcement Learning method for Epidemic control via individual-level mobility intervention, namely Vehicle. To tackle the above challenges, Vehicle first exploits an information rebuilding module that consists of a contact-risk bipartite graph neural network and a variational LSTM to restore the unobservable information. The contact-risk bipartite graph neural network estimates the possibility of an individual being an asymptomatic infection and the risk of this individual spreading the epidemic, as the current state of RL. Then, the Variational LSTM further encodes the state sequence to model the latency of epidemic spreading caused by unobservable asymptomatic infections. Finally, a Hierarchical Reinforcement Learning framework is employed to train Vehicle, which contains dual-level agents to solve the delayed reward problem. Extensive experimental results demonstrate that Vehicle can effectively control the spread of the epidemic. Vehicle outperforms the state-of-the-art baseline methods with remarkably high-precision mobility interventions on both symptomatic and asymptomatic infections. © 2022 Owner/Author.

Interpretable AI for policy-making in pandemics

Since the first wave of the COVID-19 pandemic, governments have applied restrictions in order to slow down its spreading. However, creating such policies is hard, especially because the government needs to trade-off the spreading of the pandemic with the economic losses. For this reason, several works have applied machine learning techniques, often with the help of special-purpose simulators, to generate policies that were more effective than the ones obtained by governments. While the performance of such approaches are promising, they suffer from a fundamental issue: since such approaches are based on black-box machine learning, their real-world applicability is limited, because these policies cannot be analyzed, nor tested, and thus they are not trustable. In this work, we employ a recently developed hybrid approach, which combines reinforcement learning with evolutionary computation, for the generation of interpretable policies for containing the pandemic. These policies, trained on an existing simulator, aim to reduce the spreading of the pandemic while minimizing the economic losses. Our results show that our approach is able to find solutions that are extremely simple, yet very powerful. In fact, our approach has significantly better performance (in simulated scenarios) than both previous work and government policies. © 2022 ACM.

Automated Lung Segmentation On Chest CT Using Deep Learning

A pandemic of coronavirus disease 2019 (COVID-19) devastated humanity before the end of 2019, which was triggered by severe acute respiratory syndrome (SARS), which originated in Wuhan, China. This sickness has claimed the lives of many people. The pandemic's consequences have been more severe in the world's most populous countries. Despite the fact that over a billion immunizations have been distributed to Indian citizens, the epidemic has not abated as of October 21, 2021. While certain limits are being eased, the threat of the dreaded "fourth wave"remains. In these situations, having technologies for swift disease testing and diagnosis is critical, since it allows for a much speedier process. Using a CT scan of the lungs, this research will provide vision into a model that may proficiently and correctly envisage the existence of COVID-19. © 2022 IEEE.